Divergences, surrogate loss functions and experimental design
نویسندگان
چکیده
In this paper, we provide a general theorem that establishes a correspondence between surrogate loss functions in classification and the family of f -divergences. Moreover, we provide constructive procedures for determining the f -divergence induced by a given surrogate loss, and conversely for finding all surrogate loss functions that realize a given f -divergence. Next we introduce the notion of universal equivalence among loss functions and corresponding f -divergences, and provide necessary and sufficient conditions for universal equivalence to hold. These ideas have applications to classification problems that also involve a component of experiment design; in particular, we leverage our results to prove consistency of a procedure for learning a classifier under decentralization requirements.
منابع مشابه
Information Divergence Measures and Surrogate Loss Functions
In this extended abstract, we provide an overview of our recent work on the connection between information divergence measures and convex surrogate loss functions used in statistical machine learning. Further details can be found in the technical report [7] and conference paper [6]. The class of f -divergences, introduced independently by Csiszar [4] and Ali and Silvey [1], arise in many areas ...
متن کاملOn divergences, surrogate loss functions, and decentralized detection
We develop a general correspondence between a family of loss functions that act as surrogates to 0-1 loss, and the class of Ali-Silvey or f -divergence functionals. This correspondence provides the basis for choosing and evaluating various surrogate losses frequently used in statistical learning (e.g., hinge loss, exponential loss, logistic loss); conversely, it provides a decision-theoretic fr...
متن کاملOn Information Divergence Measures, Surrogate Loss Functions and Decentralized Hypothesis Testing
We establish a general correspondence between two classes of statistical functions: AliSilvey distances (also known as f -divergences) and surrogate loss functions. Ali-Silvey distances play an important role in signal processing and information theory, for instance as error exponents in hypothesis testing problems. Surrogate loss functions (e.g., hinge loss, exponential loss) are the basis of ...
متن کاملMulticlass Classification, Information, Divergence, and Surrogate Risk
We provide a unifying view of statistical information measures, multi-class classificationproblems, multi-way Bayesian hypothesis testing, and loss functions, elaborating equivalenceresults between all of these objects. In particular, we consider a particular generalization off -divergences to multiple distributions, and we show that there is a constructive equivalencebetwee...
متن کاملConvexity, Detection, and, Generalized f-divergences
The goal of multi-class classification problem is to find a discriminant function that minimizes the expectation of 0-1 loss function. However, minimizing 0-1 loss directly is often computationally intractable and practical algorithms are usually based on convex relaxations of 0-1 loss, say Φ, which is called the surrogate loss. In many applications, the covariates are either not available dire...
متن کامل